<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
  <title>NOAA Autonomous Hydrophone Data Format</title>
</head>
<body alink="#008b8b" bgcolor="#fffff0" link="#0000ff" text="#000000"
 vlink="#8a2be2">
<h3>
<a href="http://www.noaa.gov" target="_parent"><img
 src="http://www.pmel.noaa.gov/images/noaa-trans.gif" alt="NOAA Logo"
 align="middle" height="67" width="67"></a>
<a href="http://www.noaa.gov" target="_parent">NOAA</a>/
<a href="http://www.pmel.noaa.gov/vents/home.html" target="_parent">OERD</a>
Autonomous Hydrophone Data File Description
</h3>
<b>Last Update: September 15, 2011</b>
<p><b>Data Format Type 3 for the NOAA Autonomous Hydrophone Data Files
from 1999 on</b>
</p>
<p>For each data file, the first 256 bytes are Header Information and
rest of the bytes are data. The <a href="#Header">contents of the
header</a> are shown in between the Byte Swapping and the <a
 href="#Decode">Decoding Instructions</a> sections in this document. To
read the data correctly,
</p>
<pre>
TIME_GMT = Start Date &amp; Time,
NCHAN    = Number of Channels.
SRATEHZ  = Sample Rate in hertz,
SAMPLES  = Sample Size and type,
</pre>
from the Header will be needed.
<p>TIME_GMT (from byte positions 74 to 115 of the header with 1st byte
position being 0) contains 2 dates and times.
An example of TIME_GMT will be
</p>
<pre>     100 046:19:04:03:990100 046:19:04:03:1012<br>     ^   Q-Tech Clock   ^|   On-Set Clock    |<br></pre>
Q-Tech Clock is very precise and independent.
On-Set or Real-Time Clock reads from the microprocessor's chip.
Both are synchronized with GPS time on the deployments.
Both clocks contain Year Julian Day:Hour:Minute:Second:sss(s).
Form the example above,
Year = 100 which means year 2000 and 101 will be year 2001,
Julian Date = 46, Hour = 19, Minute = 4 and Seconds in
Q-Tech is 03:990 which mean 3.990 seconds and in
On-Set is 03:1012 which mean 3 + 1012/1024 = 3.988281 seconds.
Please note that sometime the leading zeros may not be there,
<pre>i.e. TIME_GMT could be in 100 046:19:04:03:990100 46:19:4:3:1012</pre>
<p>NCHAN (from byte positions 124 to 125) is a short integer value
to indicate the how many channels per data file.  For the hydrophone data
files before 2009, there are only 1 Channel per file.  Started from 2009
on, there are hydrophone data files contain 2 Channels, e.g. 1st Channel
is for the data signals and the 2nd channel is for the 1-PPS signals
(Timing Signals in a Square-Wave form).  
</p>
<p>
SRATEHZ (from byte positions 126 to 127) is a short integer
value that indicate the Sample Rate, e.g. 1000 Hz. Note that
SRATEHZ should not be used as the true Sample Rate. The correct
Sample Rate must be computed (see Compute the Sample Rate below)
and the value should be very close to SRATEHZ.
</p>
<p>SAMPLES (from byte positions 128 to 129) is a short integer value
to indicate how the data value in the file are being store after
the header. SAMPLES can be either 0, 2 or 3 where<br>
0 indicates the data after the header are in byte values;<br>
2 indicates the data after the header are 12-bit unsigned integers
store in the short integers, i.e., 2 bytes per 1 data point and the
last 4 high order bits must be set to zeros in order to interpret
the value correctly; and<br>
3 indicates 16-bit unsigned integers are
stored into the short integers, i.e. 2 bytes per 1 data point.<br>
Also, see Byte Swapping below.
</p>
<p><b>Compute the Sample Rate:</b>
</p>
<p>Because of the hardware setup, Sample Rate must be deteremined for
each data file by the total data points in the file divided by the
total time segment in seconds.
</p>
<p>(A) Total Data Points in a given data file will be<br>
(File Size in Bytes - Header: 256 bytes )/(Data Size)/NCHAN</br> where
Data Size is 1 if SAMPLES = 0 and Data Size is 2 if SAMPLES = 2 or 3;
NCHAN >= 1.<br>
</p>
<p>(B) Total Time Segment will be the End Time - Start Time where the
Start Time can be obtained from the given data file, e.g. 000011.dat.
The End Time, however, must be retrieved from the next following
data file, e.g. 000012.dat and use its start time as the End Time.
</p>
<p>Then (A)/(B) will give the correct Sample Rate for the data file:
000011.dat. If the next data file is not available or does not
exist, SRATEHZ will need to be used.
</p>
<p><b>Byte (8-bit) Values:</b>
</p>
<p>When SAMPLES = 0, the data are 8-bit byte numbers. To get the byte
numbers, read them as unsigned 8-bit numbers: B. Then convert them
into signed integers: I = B - 127.
</p>
<p><b>Byte Swapping:</b>
</p>
<p>When SAMPLES = 2 (12-bit) or 3 (16-bit) numbers, the data are stored
into the 2 bytes short integers and are generated by a Big Endian
format,
i.e, the most-significant, left most bit or byte.
Byte Swapping for the short integers will be needed if a computer
with a litte endian format is used.&nbsp; Then the unsigned values
(UI2) need to be converted to signed integers: I2 = UI2 -&nbsp; N where
N = 2048 for 12-bit data or N = 32768 for 16-bit data.<br>
</p>
<p><b><a name="Header">Header</a> Format for Autonomous Hydrophone Data
Files Type 3:</b>
</p>
<p>The header format is shown in C programming language. There are many
redundancy, for example, RTC, Q-Tech time in seconds then GMT ASCII
format. Also more information are added since September 17, 2002. The
new information are shown in <font color="BLUE">Blue</font>.
</p>
<pre>typedef struct {<br>//<br>// ----- First 32 bytes of <a name="BIR">BIR</a> header is working variables ------<br>//<br>  char   BIRHdrID[4];  // "BIR\0"<br>  ushort BIRVersion;   // version.release * 10<br>  ushort BIRUserHeaderSize; // size in bytes for user header<br>  ushort BIRUnused;    // currently unused (zero)<br>  ulong  RTCsecs;      // current RTC seconds<br>  ushort RTCticks;     // current RTC ticks<br>//<br>  ulong BIRCapacityBytes;  // total capacity of current drive<br>  ulong BIRStartFreeBytes; // free space on current drive at start<br>  ulong BIRReceivedBytes;  // total number of bytes received (likely to wrap)<br>  ulong BIRWrittenBytes;   // total number of bytes written (likely to wrap)<br>//<br>// ----- Second 32 bytes of BIR header is copy of VEE settings ------<br>//<br>  long CFPPBSZ;     // size of CompactFlash buffer, typ. 40MB<br>  long RAMPPBSZ;    // size of data RAM PP buffer, typ. 16 to 64KB<br>  long RAMHDBFSZ;   // size of CF to HD copy buffer, typ. 16 to 64KB<br>  long MINFREESZ;   // minimum free space until switch to next drive<br>//<br>  char HDDOSDRV[4]; // DOS drive assigned to hard disk ("D:")<br>  short NODRVTEST;  // testing without accessing drive<br>  short UARTMONIT;  // sending diagnostics to RS-232 port<br>  short FLOGFLAG;   // flag to log for major events (startup, spinups)<br>  short BIADEVICE;  // device type attached to BigIDEA(s)<br>  short CURBIA;     // index to current BigIDEA/drive<br>  short CURPRTN;    // index to current DOS partition<br>}<br><br>//<br>// User header gets appended to the end of this<br>//<br><br>#define LOGF_NAME_MAX 16  // max filename length for log file<br><font
 color="BLUE">#define EXPID_LENMAX  16  // It was set to 82 before 9/17/2002.</font>

typedef struct {
// ----- Total 192 bytes header
// ----- First 168 bytes of ACQ header is copy of VEE settings ------
//
  char  <a
 name="SYSTEM">SYSTEM[3]</a>;    // system name (e.g., P01)<br>  char  LOCATION[7];  // 7-char locaiton starting lat followed by long<br>  char  TIME_GMT[42]; // GMT time<br>  char  EXPID[EXPID_LENMAX];    // experiment ID<br><font
 color="BLUE">
  char  PROGNAME[16]; // Program Name.           Added on 09/22/2002.
  long  CF1SERNUM;    // CF1 Serial Number.      Added on 10/01/2002.
  char  TABLE[4];     // Pre-amp gain table.     Added on 10/01/2002.
  char  PRMNAME[10];  // Pre-amp Name.           Added on 09/22/2002.
  char  DAQNAME[10];  // DAQ board Name.         Added on 09/22/2002.
  char  NUMDISK[8];   // Dsiak system.           Added on 09/22/2002.
  char  CLK[10];      // Clock board name.       Added on 09/22/2002.
  short PWFILT;       // Pre-Whitening Filte setting 1,2,3,4.  Added on 09/17/2002.
  short LOPASS;       // Low Pass cut-off [Hz].  Added on 09/17/2002.
</font>
  ulong WAKEUP;       // time to start logging in seconds since 01/01/1970.  <!--  ulong WAKEUP;       // time to start logging 11/02/99 NOAA hm -->
  char  LOGFILE[LOGF_NAME_MAX]; // filename for event logging
  short STARTUPS;     // number of startups we've seen
  short MAXSTRTS;     // maximum allowable startups
  short NCHAN;        // number of analog channels
  short SRATEHZ;      // sample rate in hertz
  short SAMPLES;      // sample size and type
  short ONFINSUSP;    // enter suspend mode on finish
  short LOGTOCON;     // flag to log events to console
//
// ----- Plus 2 bytes for version number
//
  ushort ACQVersion;  // version.release * 10
//
// ----- Last 22 bytes of BIR header is your working variables ------
// THIS SHOULD BE CUSTOMIZED TO YOUR APPLICATIONS REQUIREMENTS
//  These are just suggestions and none are essential to the program
//
  ulong OnePPSCounter;      // user supplied 1pps Q-Tech count
  ulong HundredPPSCounter;  // user supplied 100pps Q-Tech count
  ulong ADTimingRuptCount;  // total convert interrupts (will wrap)
  ulong ADSamlingRuptCount; // total A-D interrupts (will wrap)
//
//*JHG990922--REMOVE LINE---  char unused[16]; // pad this out for 32 byte total
//
  short SLEEP;     // added 8/27/99 by hm
  short GAIN;      // additional pre-amp gain 0 to 3 with 6dB increm
  char unused[2];  // *JHG990922--ADD LINE--- // pad this out for 32 byte total
}
</pre>
<p>
<b><a name="Decode">Instructions</a> for Decoding Autonomous Hydrophone
Data Files</b>
</p>
<p>The following instructions are one of many ways to decode the data
file.
The objective is to give users examples for reading the header and
data.
Note that the following codes only read the essential information from
the header in order to compute the sample rate and read the data
correctly.
</p>
<p><b>Decoding Autonomous Hydrophone Data (Format 2) Block using
<a href="http://www.rsinc.com/idl/"
 onmouseout="window.status=''; return true;"
 onmouseover="window.status='Interactive Data Language from Research Systems, Inc.'; return true;">IDL</a>
Software Package</b>
</p>
<p>The following IDL commands will decode an autonomous data file 1 block
at a time. Note that this section is assuming readers know IDL
and how to refer to the IDL manual for detail.
</p>
<p>Enter IDL ( in UNIX system, type IDL then IDL&gt; should show up )</p>
<pre>
IDL&gt; OPENR, 10, 'Type3Format.data'     ; Type3Format.data is the file name.
IDL&gt; BLK = ASSOC( 10, BYTARR( 256 ) )  ; 256 is the block size in bytes
                                       ; and it is also the Header size.
;    BLK will allow users to get 256 bytes of data per read.

IDL&gt; HDR = BLK[0]  ; Read in the Header: HDR which will be 1-D byte array
                   ; with 256 elements from 0 to 255.

;    Show the <a href="#SYSTEM">SYSTEM, LOCATION &amp; TIME_GMT</a>.
IDL&gt; PRINT, 'SYSTEM[3]   : ', STRING( HDR[064:066] )
IDL&gt; PRINT, 'LOCATION[7] : ', STRING( HDR[067:073] )
IDL&gt; PRINT, 'TIME_GMT[42]: ', STRING( HDR[074:115] )

;    Decode the 7 short intergers for <a href="#SYSTEM">STARTUPS, MAXSTRTS, NCHANNALS</a>,
;    <a href="#SYSTEM">SRATEHZ (SampleRate), SAMPLES Size, Type, ONFINSUSP, &amp; LOGTOCON</a>.
IDL&gt; INTEGER2 =  FIX( HEADER, EXPID_LEN_MAX+LOGF_NAME_MAX+120, 7 )
;    where EXPID_LEN_MAX = 16 and LOGF_NAME_MAX = 82 see the Header Format above.
;    120 = all the bytes from 0 to the last byte position of LOGFILE[LOGF_NAME_MAX].

;    If your computer is using Litte Endian Format, you must do the Byte Swapping.
IDL&gt; BYTEORDER, INTEGER2, /SSWAP  ; Skip this if your computer is Big Endian.

IDL&gt; PRINT, 'STARTUPS          : ',  INTEGER2[0]
IDL&gt; PRINT, 'MAXSTRTS          : ',  INTEGER2[1]
IDL&gt; PRINT, 'NCHANnals         : ',  INTEGER2[2]
IDL&gt; PRINT, 'SRATEHZ SampleRate: ',  INTEGER2[3]
IDL&gt; PRINT, 'SAMPLES Size, Type: ',  INTEGER2[4]
IDL&gt; PRINT, 'ONFINSUSP         : ',  INTEGER2[5]
IDL&gt; PRINT, 'LOGTOCON          : ',  INTEGER2[6]

;    Save the Number of Channels (NCHAN) per file.
IDL&gt; NCHAN = INTEGER2[2]  ; It will be used below.

;    Decode data blocks.  <b>Assuming SAMPLE Size Type is 0, i.e. 1-byte data.</b>
IDL&gt; B   = BLK[1]   ; Read in the 2nd block, the 2nd set of 256 bytes.
IDL&gt; RCD = B - 127  ; Convert the data into the range between -127 to 128.
;    where RCD will be an integer array of 256 elements.

;    If NCHAN == 1, then RCD contains all the data for the 1st and only channel.
IDL&gt; DATA = TEMPORARY( RCD )  ; Optional: Assign RCD into DATA at 1-D array.

;    If NCHAN >= 2, then RCD must be converted in to a 2-D array as NCHAN x N
;    where N = Total data points per block = 256/NCHAN.
IDL&gt; DATA = FIX( RCD, 0, NCHAN, 256/NCHAN )  ; DATA = 2 x 128 assuming NCHAN = 2.
;    then DATA[0,*] = DATA[0,0:256/NCHAN-1]  ; Data Signal from the 1st Channel.
;    and  DATA[1,*] = DATA[1,0:256/NCHAN-1]  ; Data Signal from the 2nd Channel.

;    Decode the next data block.
IDL&gt; B   = BLK[2]  &amp;  RCD = B - 127  ; Process 2 IDL statements together.

;    Note that the order of which block to read can be random, i.e.
;    B = BLK[10] &amp; ...  and later B = BLK[5] &amp; ... are OK.

     ::: etc :::

;    Decode data blocks with <b>SAMPLE Size Type = 3 (Short Integer) or 2 (12-Bits Integer).</b>
IDL&gt; B  = BLK[1]            ; Read in the 2nd set of 256 bytes from the data file.
IDL&gt; I2 = UINT( B, 0, 128 ) ; convert 256 bytes into 2-bytes unsigned integers
                            ; with 128 elements.
IDL&gt; BYTEORDER, I2, /SSWAP  ; <b>Required if Litte Endian computer is used.</b>
IDL&gt; I2 = I2 - 2048         ; Convert I2 to a 2-bytes signed integer array.

;    If NCHAN == 1, then I2 contains all the data for the 1st and only channel.
IDL&gt; DATA = TEMPORARY( I2 )  ; Optional: Assign I2 into DATA at 1-D array.

;    If NCHAN >= 2, then I2 must be converted in to a 2-D array as NCHAN x N
;    where N = Total data points per block = 128/NCHAN.
IDL&gt; DATA = FIX( I2, 0, NCHAN, 128/NCHAN )  ; DATA = 2 x 64 assuming NCHAN = 2.
;    then DATA[0,*] = DATA[0,0:128/NCHAN-1]  ; Data Signal from the 1st Channel.
;    and  DATA[1,*] = DATA[1,0:128/NCHAN-1]  ; Data Signal from the 2nd Channel.

     ::: etc :::

IDL&gt; CLOSE, 10  ; Close the data file when you are done.
IDL&gt; EXIT
</pre>
<p>
<b>Decoding Autonomous Data Block using
<a href="http://www.mathworks.com/"
 onmouseout="window.status=''; return true;"
 onmouseover="window.status='Interactive Language from the MathWorks Company.'; return true;">MatLab</a>
Software Package</b>
</p>
<p>The following MatLab commands will decode an autonomous data file 1 block
at a time. Note that this section is assuming readers know MatLab
and how to refer to the MatLab manual for detail.
</p>
<p>Note that the Matlab fopen() function will read the Big Endian format
to binary the data as the default (however the 'b' option still should be used
see the (1) below); therefore, no byteswapping is needed.
</p>
<p>Enter MatLab ( in UNIX system, type MatLab then the MatLab working
window will show up )
</p>
<pre>    Open a data file (1) and Skip the 1st 64 bytes
    (<a href="#BIR">BIR</a> working and VEE setting variables) from the file (2).
&gt;&gt;&gt; FID    = fopen( 'Type3FormatDataFileName', 'r', 'b' );  % (1)
&gt;&gt;&gt; status = fseek( FID, 64, 'bof' );                       % (2)
Read in the next 52 bytes for the <a href="#SYSTEM">SYSTEM, LOCATION &amp; TIME_GMT</a>.
&gt;&gt;&gt; a      = fread( FID, 52 );
&gt;&gt;&gt; disp( char( a(1:3)'   ) )  % Show the SYSTEM.
    H19<br>&gt;&gt;&gt; disp( char( a(4:10)'  ) )  % Show LOCATION.
    N50W145<br>&gt;&gt;&gt; disp( char( a(11:52)' ) )  % Show TIME_GMT.
    101 093:18:19:37:710101 093:18:19:37:0726

    Skip the 1st 218 bytes from the beginning of the file, and
    Read in 7 short intergers for <a href="#SYSTEM">STARTUPS, MAXSTRTS, NCHANNALS</a>,
    <a href="#SYSTEM">SRATEHZ (SampleRate), SAMPLES Size, Type, ONFINSUSP, &amp; LOGTOCON</a>.
&gt;&gt;&gt; status = fseek( FID, 218, 'bof' );    % Move to the correct positon.
&gt;&gt;&gt; i      = fread( FID, 7, 'short' );    % Read in 7 short intergers
&gt;&gt;&gt; i(3)   % Shows the NCHAN (Total Channels)which is the 3rd elements in i.
&gt;&gt;&gt; i(4)   % Shows the SRATEHZ (Sample Rate) which is the 4th elements in i.
&gt;&gt;&gt; i(5)   % Shows the Sample Size           which is the 5th elements in i.
&gt;&gt;&gt; nchan  = i(3);  % Save the Total Channels & it will be used below.

    <b>Assuming SAMPLE Size Type is 0, i.e. 1-byte data</b>
    Decode data blocks as nchan x (pts = total data points per channel).
&gt;&gt;&gt; status = fseek( FID, 256, 'bof'   );  % Skip the 1st 256 header bytes.
&gt;&gt;&gt; pts    = 256/nchan;                   % Number of bytes per channel to be read.
&gt;&gt;&gt; blk    = fread( FID, [ nchan pts ], 'uchar' );  % Read in the data as after the header.
&gt;&gt;&gt; rcd    = blk - 127;                   % convert the range into -127 &amp; 128.

    Decode the next data block.
&gt;&gt;&gt; blk    = fread( FID, [ nchan pts ], 'uchar' );  % Read in the next data set.
&gt;&gt;&gt; rcd    = blk - 127;                   % convert the range into -127 &amp; 128.

    ::: etc :::

    Note that you could set variable: pts to any size &lt;= Max. (data) bytes in
    the input file, i.e. pts = ((data file size) - 256)/nchan, if your computer
    heas enough memory to hold all the data.

    <b>Assuming SAMPLE Size Type is 2 or 3, i.e. 12-bit or 16-bit data</b>
    Decode data blocks as nchan x (pts = total data points per channel).
&gt;&gt;&gt; status = fseek( FID, 256, 'bof'   );  % Skip the 1st 256 header bytes.
&gt;&gt;&gt; pts    = 256/nchan/2;                 % Number of bytes per channel to be read.
&gt;&gt;&gt; blk    = fread( FID, [ nchan inf ], 'uint16' );  % Read in ALL the data after the header.
&gt;&gt;&gt; rcd    = blk' - 32768;  % convert the range into -32768 &amp; 32767.

    ::: etc :::

    where rcd is a 2-D array.  Assumming nchan == 2 then
    rcd(:,1) = data signals from the 1st channel, and
    rcd(:,2) = data signals from the 2nd channel.

</pre>
<b>Other Data File Formats</b>
<p>Data Format <a href="./HPdataFormat2.html"> Type 2</a>. Note that
Data Format Type 1 was used for experiment and it is No longer being
used.
<!-- P>
<B>T-Phase data files are available to be 
<A HREF="./TPDretrieval.html" onMouseOut="window.status=''; return true;"
 onMouseover="window.status='To T-Phase Data Retrieval Center.';return true;">
retrieved</A>.</B --></p>
</body>
</html>
